Time-Series Aware Metrics for the Evaluation of Intraoperative Electroencephalography-Based Ischemia Detection.

Continuous intraoperative monitoring with electroencephalo2 graphy (EEG) is commonly used to detect cerebral ischemia in high-risk surgical procedures such as carotid endarterectomy. Machine learning (ML) models that detect ischemia in real time can form the basis of automated intraoperative EEG monitoring. In this study, we describe and compare two time-series aware precision and recall metrics to the classical precision and recall metrics for evaluating the performance of ML models that detect ischemia. We trained six ML models to detect ischemia in intraoperative EEG and evaluated them with the area under the precision-recall curve (AUPRC) using time-series aware and classical approaches to compute precision and recall. The Support Vector Classification (SVC) model performed the best on the time-series aware metrics, while the Light Gradient Boosting Machine (LGBM) model performed the best on the classical metrics. Visual inspection of the probability outputs of the models alongside the actual ischemic periods revealed that the time-series aware AUPRC selected a model more likely to predict ischemia onset in a timely fashion than the model selected by classical AUPRC.

Design and implementation of a clinical laboratory information system in a low-resource setting.

BACKGROUND: Reducing laboratory errors presents a significant opportunity for both cost reduction and healthcare quality improvement. This is particularly true in low-resource settings where laboratory errors are further exacerbated by poor infrastructure and shortages in a trained workforce. Informatics interventions can be used to address some of the sources of laboratory errors. OBJECTIVES: This article describes the development process for a clinical laboratory information system (LIS) that leverages informatics interventions to address problems in the laboratory testing process at a hospital in a low-resource setting. METHODS: We designed interventions using informatics methods for previously identified problems in the laboratory testing process at a clinical laboratory in a low-resource setting. First, we reviewed a pre-existing LIS functionality assessment toolkit and consulted with laboratory personnel. This provided requirements that were developed into a LIS with interventions designed to address the problems that had been identified. We piloted the LIS at the Kamuzu Central Hospital in Lilongwe, Malawi. RESULTS: We implemented a series of informatics interventions in the form of a LIS to address sources of laboratory errors and support the entire laboratory testing process. Custom hardware was built to support the ordering of laboratory tests and review of laboratory test results. CONCLUSION: Our experience highlights the potential of using informatics interventions to address systemic problems in the laboratory testing process in low-resource settings. Implementing these interventions may require innovation of new hardware to address various contextual issues. We strongly encourage thorough testing of such innovations to reduce the risk of failure when implemented.

Design and implementation of a clinical laboratory information system in a low-resource setting

Background: Reducing laboratory errors presents a significant opportunity for both cost reduction and healthcare quality improvement. This is particularly true in low-resource settings where laboratory errors are further exacerbated by poor infrastructure and shortages in a trained workforce. Informatics interventions can be used to address some of the sources of laboratory errors.Objectives: This article describes the development process for a clinical laboratory information system (LIS) that leverages informatics interventions to address problems in the laboratory testing process at a hospital in a low-resource setting.Methods: We designed interventions using informatics methods for previously identified problems in the laboratory testing process at a clinical laboratory in a low-resource setting. First, we reviewed a pre-existing LIS functionality assessment toolkit and consulted with laboratory personnel. This provided requirements that were developed into a LIS with interventions designed to address the problems that had been identified. We piloted the LIS at the Kamuzu Central Hospital in Lilongwe, Malawi.Results: We implemented a series of informatics interventions in the form of a LIS to address sources of laboratory errors and support the entire laboratory testing process. Custom hardware was built to support the ordering of laboratory tests and review of laboratory test results.Conclusion: Our experience highlights the potential of using informatics interventions to address systemic problems in the laboratory testing process in low-resource settings. Implementing these interventions may require innovation of new hardware to address various contextual issues. We strongly encourage thorough testing of such innovations to reduce the risk of failure when implemented

User-centered design and usability testing of RxMAGIC: a prescription management and general inventory control system for free clinic dispensaries.

BACKGROUND: To address challenges related to medication management in underserved settings, we developed a system for Prescription Management And General Inventory Control, or RxMAGIC, in collaboration with the Birmingham Free Clinic in Pittsburgh, Pennsylvania. RxMAGIC is an interoperable, web-based medication management system designed to standardize and streamline the dispensing practice and improve inventory control in a free clinic setting. This manuscript describes the processes used to design, develop, and deploy RxMAGIC. METHODS: We transformed data from previously performed mixed-methods needs assessment studies into functional user requirements using agile development methods. Requirements took the form of user stories that were prioritized to drive implementation of RxMAGIC as a web-application. A functional prototype was developed and tested to understand its perceived usefulness before developing a production system. Prior to deployment, we evaluated the usability of RxMAGIC with six users to diagnose potential interaction challenges that may be avoided through redesign. The results from this study were similarly prioritized and informed the final features of the production system. RESULTS: We developed 45 user stories that acted as functional requirements to incrementally build RxMAGIC. Integrating with the electronic health record at the clinic was a requirement for deployment. We utilized health data standards to communicate with the existing order entry system; an outgoing electronic prescribing framework was leveraged to send prescription data to RxMAGIC. The results of the usability study were positive, with all tested features receiving a mean score of four or five (i.e. somewhat easy or easy, respectively) on a five-point Likert scale assessing ease of completion, thus demonstrating the system's simplicity and high learnability. RxMAGIC was deployed at the clinic in October 2016 over a two-week period. CONCLUSIONS: We built RxMAGIC, an open-source, pharmacist-facing dispensary management information system that augments the pharmacist's ability to efficiently deliver medication services in a free clinic setting. RxMAGIC provides electronic dispensing and automated inventory management and alerting capabilities. We deployed RxMAGIC at the Birmingham Free Clinic and measured its usability with potential users. In future work, we plan to continue to measure the impact of RxMAGIC on pharmacist efficiency and satisfaction.

The center for causal discovery of biomedical knowledge from big data.

The Big Data to Knowledge (BD2K) Center for Causal Discovery is developing and disseminating an integrated set of open source tools that support causal modeling and discovery of biomedical knowledge from large and complex biomedical datasets. The Center integrates teams of biomedical and data scientists focused on the refinement of existing and the development of new constraint-based and Bayesian algorithms based on causal Bayesian networks, the optimization of software for efficient operation in a supercomputing environment, and the testing of algorithms and software developed using real data from 3 representative driving biomedical projects: cancer driver mutations, lung disease, and the functional connectome of the human brain. Associated training activities provide both biomedical and data scientists with the knowledge and skills needed to apply and extend these tools. Collaborative activities with the BD2K Consortium further advance causal discovery tools and integrate tools and resources developed by other centers.

A method for detecting and characterizing outbreaks of infectious disease from clinical reports.

Outbreaks of infectious disease can pose a significant threat to human health. Thus, detecting and characterizing outbreaks quickly and accurately remains an important problem. This paper describes a Bayesian framework that links clinical diagnosis of individuals in a population to epidemiological modeling of disease outbreaks in the population. Computer-based diagnosis of individuals who seek healthcare is used to guide the search for epidemiological models of population disease that explain the pattern of diagnoses well. We applied this framework to develop a system that detects influenza outbreaks from emergency department (ED) reports. The system diagnoses influenza in individuals probabilistically from evidence in ED reports that are extracted using natural language processing. These diagnoses guide the search for epidemiological models of influenza that explain the pattern of diagnoses well. Those epidemiological models with a high posterior probability determine the most likely outbreaks of specific diseases; the models are also used to characterize properties of an outbreak, such as its expected peak day and estimated size. We evaluated the method using both simulated data and data from a real influenza outbreak. The results provide support that the approach can detect and characterize outbreaks early and well enough to be valuable. We describe several extensions to the approach that appear promising.

PaTH: towards a learning health system in the Mid-Atlantic region.

The PaTH (University of Pittsburgh/UPMC, Penn State College of Medicine, Temple University Hospital, and Johns Hopkins University) clinical data research network initiative is a collaborative effort among four academic health centers in the Mid-Atlantic region. PaTH will provide robust infrastructure to conduct research, explore clinical outcomes, link with biospecimens, and improve methods for sharing and analyzing data across our diverse populations. Our disease foci are idiopathic pulmonary fibrosis, atrial fibrillation, and obesity. The four network sites have extensive experience in using data from electronic health records and have devised robust methods for patient outreach and recruitment. The network will adopt best practices by using the open-source data-sharing tool, Informatics for Integrating Biology and the Bedside (i2b2), at each site to enhance data sharing using centrally defined common data elements, and will use the Shared Health Research Information Network (SHRINE) for distributed queries across the network.

Influenza detection from emergency department reports using natural language processing and Bayesian network classifiers.

OBJECTIVES: To evaluate factors affecting performance of influenza detection, including accuracy of natural language processing (NLP), discriminative ability of Bayesian network (BN) classifiers, and feature selection. METHODS: We derived a testing dataset of 124 influenza patients and 87 non-influenza (shigellosis) patients. To assess NLP finding-extraction performance, we measured the overall accuracy, recall, and precision of Topaz and MedLEE parsers for 31 influenza-related findings against a reference standard established by three physician reviewers. To elucidate the relative contribution of NLP and BN classifier to classification performance, we compared the discriminative ability of nine combinations of finding-extraction methods (expert, Topaz, and MedLEE) and classifiers (one human-parameterized BN and two machine-parameterized BNs). To assess the effects of feature selection, we conducted secondary analyses of discriminative ability using the most influential findings defined by their likelihood ratios. RESULTS: The overall accuracy of Topaz was significantly better than MedLEE (with post-processing) (0.78 vs 0.71, p<0.0001). Classifiers using human-annotated findings were superior to classifiers using Topaz/MedLEE-extracted findings (average area under the receiver operating characteristic (AUROC): 0.75 vs 0.68, p=0.0113), and machine-parameterized classifiers were superior to the human-parameterized classifier (average AUROC: 0.73 vs 0.66, p=0.0059). The classifiers using the 17 'most influential' findings were more accurate than classifiers using all 31 subject-matter expert-identified findings (average AUROC: 0.76>0.70, p<0.05). CONCLUSIONS: Using a three-component evaluation method we demonstrated how one could elucidate the relative contributions of components under an integrated framework. To improve classification performance, this study encourages researchers to improve NLP accuracy, use a machine-parameterized classifier, and apply feature selection methods.

Association of over-the-counter pharmaceutical sales with influenza-like-illnesses to patient volume in an urgent care setting.

We studied the association between OTC pharmaceutical sales and volume of patients with influenza-like-illnesses (ILI) at an urgent care center over one year. OTC pharmaceutical sales explain 36% of the variance in the patient volume, and each standard deviation increase is associated with 4.7 more patient visits to the urgent care center (p<0.0001). Cross-correlation function analysis demonstrated that OTC pharmaceutical sales are significantly associated with patient volume during non-flu season (p<0.0001), but only the sales of cough and cold (p<0.0001) and thermometer (p<0.0001) categories were significant during flu season with a lag of two and one days, respectively. Our study is the first study to demonstrate and measure the relationship between OTC pharmaceutical sales and urgent care center patient volume, and presents strong evidence that OTC sales predict urgent care center patient volume year round.

A method for estimating from thermometer sales the incidence of diseases that are symptomatically similar to influenza.

Early detection and accurate characterization of disease outbreaks are important tasks of public health. Infectious diseases that present symptomatically like influenza (SLI), including influenza itself, constitute an important class of diseases that are monitored by public-health epidemiologists. Monitoring emergency department (ED) visits for presentations of SLI could provide an early indication of the presence, extent, and dynamics of such disease in the population. We investigated the use of daily over-the-counter thermometer-sales data to estimate daily ED SLI counts in Allegheny County (AC), Pennsylvania. We found that a simple linear model fits the data well in predicting daily ED SLI counts from daily counts of thermometer sales in AC. These results raise the possibility that this model could be applied, perhaps with adaptation, in other regions of the country, where commonly thermometer sales data are available, but daily ED SLI counts are not.

Probabilistic case detection for disease surveillance using data in electronic medical records.

This paper describes a probabilistic case detection system (CDS) that uses a Bayesian network model of medical diagnosis and natural language processing to compute the posterior probability of influenza and influenza-like illness from emergency department dictated notes and laboratory results. The diagnostic accuracy of CDS for these conditions, as measured by the area under the ROC curve, was 0.97, and the overall accuracy for NLP employed in CDS was 0.91.

Probabilistic, Decision-theoretic Disease Surveillance and Control.

The Pittsburgh Center of Excellence in Public Health Informatics has developed a probabilistic, decision-theoretic system for disease surveillance and control for use in Allegheny County, PA and later in Tarrant County, TX. This paper describes the software components of the system and its knowledge bases. The paper uses influenza surveillance to illustrate how the software components transform data collected by the healthcare system into population level analyses and decision analyses of potential outbreak-control measures.

Key design elements of a data utility for national biosurveillance: event-driven architecture, caching, and Web service model.

The National Retail Data Monitor (NRDM) has monitored over-the-counter (OTC) medication sales in the United States since December 2002. The NRDM collects data from over 18,600 retail stores and processes over 0.6 million sales records per day. This paper describes key architectural features that we have found necessary for a data utility component in a national biosurveillance system. These elements include event-driven architecture to provide analyses of data in near real time, multiple levels of caching to improve query response time, high availability through the use of clustered servers, scalable data storage through the use of storage area networks and a web-service function for interoperation with affiliated systems. The methods and architectural principles are relevant to the design of any production data utility for public health surveillance-systems that collect data from multiple sources in near real time for use by analytic programs and user interfaces that have substantial requirements for time-series data aggregated in multiple dimensions.

The RODS Open Source Project: removing a barrier to syndromic surveillance.

The goal of the Real-time Outbreak and Disease Surveillance (RODS) Open Source Project is to accelerate deployment of computer-based syndromic surveillance. To this end, the project has released the RODS software under the GNU General Public License and created an organizational structure to catalyze its development. This paper describes the design of the software, requested extensions, and the structure of the development effort.

Removing a barrier to computer-based outbreak and disease surveillance--the RODS Open Source Project.

INTRODUCTION: Computer-based outbreak and disease surveillance requires high-quality software that is well-supported and affordable. Developing software in an open-source framework, which entails free distribution and use of software and continuous, community-based software development, can produce software with such characteristics, and can do so rapidly. OBJECTIVES: The objective of the Real-Time Outbreak and Disease Surveillance (RODS) Open Source Project is to accelerate the deployment of computer-based outbreak and disease surveillance systems by writing software and catalyzing the formation of a community of users, developers, consultants, and scientists who support its use. METHODS: The University of Pittsburgh seeded the Open Source Project by releasing the RODS software under the GNU General Public License. An infrastructure was created, consisting of a website, mailing lists for developers and users, designated software developers, and shared code-development tools. These resources are intended to encourage growth of the Open Source Project community. Progress is measured by assessing website usage, number of software downloads, number of inquiries, number of system deployments, and number of new features or modules added to the code base. RESULTS: During September--November 2003, users generated 5,370 page views of the project website, 59 software downloads, 20 inquiries, one new deployment, and addition of four features. CONCLUSIONS: Thus far, health departments and companies have been more interested in using the software as is than in customizing or developing new features. The RODS laboratory anticipates that after initial installation has been completed, health departments and companies will begin to customize the software and contribute their enhancements to the public code base.

Automated syndromic surveillance for the 2002 Winter Olympics.

The 2002 Olympic Winter Games were held in Utah from February 8 to March 16, 2002. Following the terrorist attacks on September 11, 2001, and the anthrax release in October 2001, the need for bioterrorism surveillance during the Games was paramount. A team of informaticists and public health specialists from Utah and Pittsburgh implemented the Real-time Outbreak and Disease Surveillance (RODS) system in Utah for the Games in just seven weeks. The strategies and challenges of implementing such a system in such a short time are discussed. The motivation and cooperation inspired by the 2002 Olympic Winter Games were a powerful driver in overcoming the organizational issues. Over 114,000 acute care encounters were monitored between February 8 and March 31, 2002. No outbreaks of public health significance were detected. The system was implemented successfully and operational for the 2002 Olympic Winter Games and remains operational today.

Design of a national retail data monitor for public health surveillance.

The National Retail Data Monitor receives data daily from 10,000 stores, including pharmacies, that sell health care products. These stores belong to national chains that process sales data centrally and utilize Universal Product Codes and scanners to collect sales information at the cash register. The high degree of retail sales data automation enables the monitor to collect information from thousands of store locations in near to real time for use in public health surveillance. The monitor provides user interfaces that display summary sales data on timelines and maps. Algorithms monitor the data automatically on a daily basis to detect unusual patterns of sales. The project provides the resulting data and analyses, free of charge, to health departments nationwide. Future plans include continued enrollment and support of health departments, developing methods to make the service financially self-supporting, and further refinement of the data collection system to reduce the time latency of data receipt and analysis.

Technical description of RODS: a real-time public health surveillance system.

This report describes the design and implementation of the Real-time Outbreak and Disease Surveillance (RODS) system, a computer-based public health surveillance system for early detection of disease outbreaks. Hospitals send RODS data from clinical encounters over virtual private networks and leased lines using the Health Level 7 (HL7) message protocol. The data are sent in real time. RODS automatically classifies the registration chief complaint from the visit into one of seven syndrome categories using Bayesian classifiers. It stores the data in a relational database, aggregates the data for analysis using data warehousing techniques, applies univariate and multivariate statistical detection algorithms to the data, and alerts users of when the algorithms identify anomalous patterns in the syndrome counts. RODS also has a Web-based user interface that supports temporal and spatial analyses. RODS processes sales of over-the-counter health care products in a similar manner but receives such data in batch mode on a daily basis. RODS was used during the 2002 Winter Olympics and currently operates in two states-Pennsylvania and Utah. It has been and continues to be a resource for implementing, evaluating, and applying new methods of public health surveillance.

Telephone triage: a timely data source for surveillance of influenza-like diseases.

We evaluated telephone triage (TT) data for public health early warning systems. TT data is electronically available and contains coded elements that include the demographics and description of a caller's medical complaints. In the study, we obtained emergency room TT data and after hours TT data from a commercial TT software and service company. We compared the timeliness of the TT data with influenza surveillance data from the Centers for Disease Control using the cross correlation function. Emergency room TT calls are one to five weeks ahead of surveillance data collected by the CDC.

Rapid deployment of an electronic disease surveillance system in the state of Utah for the 2002 Olympic Winter Games.

The key to minimizing the effects of an intentionally caused disease outbreak is early detection of the attack and rapid identification of the affected individuals. The Bush administration's leadership in advocating for biosurveillance systems capable of monitoring for bioterrorism attacks suggests that we should move quickly to establish a nationwide early warning biosurveillance system as a defense against this threat. The spirit of collaboration and unity inspired by the events of 9-11 and the 2002 Olympic Winter Games in Salt Lake City provided the opportunity to demonstrate how a prototypic biosurveillance system could be rapidly deployed. In seven weeks we were able to implement an automated, real-time disease outbreak detection system in the State of Utah and monitored 80,684 acute care visits occurring during a 28-day period spanning the Olympics. No trends of immediate public health concern were identified.